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Abstract 



In the local, characteristic 0, non archimedean case, we consider 
distributions on GL{n+l) which are invariant under the adjoint action 
of GL{n). We prove that such distributions are invariant by transpo- 
sition. This implies that an admissible irreducible representation of 
GL(n + 1), when restricted to GL{n) decomposes with multiplicity 
one. 

Similar Theorems are obtained for orthogonal or unitary groups. 



Let F be a local field non archimedean and of characteristic 0. Let W be 
a vector space over F of finite dimension ri + 1 ^ 1 and lei W = V ® U he a. 
direct sum decomposition with dimy = n. Then we have an imbedding of 
GL{y) into GL{W). Our goal is to prove the following Theorem: 

Theorem 1: If vr (resp. p) is an irreducible admissible representation of 



We choose a basis of V and a non zero vector in U thus getting a basis of 
W. We can identify GL{W) with GL{n + 1,F) and GL{V) with GL{n,¥). 



Introduction 



GL{W) (resp. ofGL{V)) then 



dim {Rome L(v)i^GL(v) , p)) ^ 1- 
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The transposition map is an involutive anti- automorphism of GL{n + l.F) 
which leaves GL{n, F) stable. It acts on the space of distributions on GL{n + 
1,F). 

Theorem 1 is a Corollary of : 

Theorem 2: A distribution on GL{W) which is invariant under the adjoint 
action of GL{V) is invariant by transposition. 

One can raise a similar question for orthogonal and unitary groups. Let 
D be either F or a quadratic extension of F. If x e D then x is the conjugate 
of X if D 7^ F and is equal to x if D = F. 

Let 1^ be a vector space over D of finite dimension n + 1 ^ 1. Let (., .) 
be a non degenerate hermitian form on W. This form is bi-additive and 

{dw, d'w') — d d'{w, w'), {w', w) — {w, w'). 

Given a D— linear map u from W into itself, its adjoint u* is defined by the 
usual formula 

{u{w),w') — {w,u*{w')) . 

Choose a vector e in 1^ such that (e, e) ^ 0; let U — 'De and V — U-^ the 
orthogonal complement. Then V has dimension n and the restriction of the 
hermitian form to V is non degenerate. 

Let M be the unitary group of W, that is to say the group of all D— linear 
maps m oi W into itself which preserve the hermitian form or equivalently 
such that mm* — 1. Let G be the unitary group of V. With the p-adic 
topology both groups are of type Ictd (locally compact, totally discontinuous) 
and countable at infinity. They arc reductive groups of classical type. 

The group G is naturally imbedded into M. 

Theorem 1': If tt (resp p) is an irreducible admissible representation of M 
(resp ofG) then 

dim (HomG(7r|G,p)) < 1- 
Choose a basis ei, . . . e„ of V such that (ej, ej) e F. For 

n 

W = XqC + ^ XiCi 
1 
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put 

n 

1 

If is a D— linear map from W into itself, let u be defined by 

u{w) = u{w). 

Let a be the anti-involution a{m) — fn~^ of M; Theorem 1' is a conse- 
quence of 

Theorem 2': A distribution on M which is invariant under the adjoint 
action of G is invariant under a. 

Let us describe briefly our proof. In section 2 we recall why Theorem 2 
(2') implies Theorem l(l'). 

Then we proceed with GL(n). The proof is by induction on n; the case 
n = is trivial. In general we first linearize the problem by replacing the 
action of G on GL(W^) by the action on the Lie algebra of GL(W^). As a 
G— module this Lie algebra is isomorphic to a direct sum g © V © © F 
with the Lie algebra of G, V* the dual space of V . The group G = GL{V) 
acts trivially on F, by the adjoint action on its Lie algebra and the natural 
actions on V and V*. The component F plays no role. Let u he a, linear 
bijection of V onto V* which transforms some basis of V into its dual basis. 
The involution may be taken as 

(X,v,v*) ^ (u-^^Xu,u-\v*),u(v)). 

We have to show that a distribution T on q(BV (BV* which is invariant under 
G and skew relative to the involution is 0. 

In section 2 we prove that such a distribution must have a singular sup- 
port. On the Q side, using Harish-Chandra descent we get that the support 
of T must be contained in ^ x J\f x {V Q) V*) where 3 is the center of g and 
J\f the cone of nilpotent elements in g. On the V (BV* side we show that the 
support must be contained in g x F where F is the cone {v, v*) = in V(BV*. 
On 3 the action is trivial so we are reduced to the case of a distribution on 

j\fxr. 

In section 3 we consider such distributions. The end of the proof is 
based on two remarks. First, viewing the distribution as a distribution on 
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A/" X {V (BV*) its partial Fourier transform relative to © V^* has the same 
invariance properties and hence must also be supported by A/" x F. This 
implies in particular a homogeneity condition on V" © V"*. The idea of using 
Fourier transform in this kind of situation goes back at least to Harish- 
Chandra ( |Ha] ) and is conveniently expressed using a particular case of the 
Weil or oscillator representation. 

For (f , f *) G F, let be the map x i-^ {x,v*)v of V into itself. The 

second remark is that the one parameter group of transformations 

is a group of (non linear) homeomorphisms of [q,q] x F which commute 
with G and the involution. It follows that the image of the support of our 
distribution must also be singular. Precisely this allows us to replace the 
condition {v,v*) = by the stricter condition X^^^* G Im adX. 

Using the stratification of M we proceed one nilpotent orbit at a time, 
transferring the problem to V^©V^* and a fixed nilpotent matrix X. The sup- 
port condition turns out to be compatible with direct sum so that it is enough 
to consider the case of a principal nilpotent element. In this last situation 
the key is the homogeneity condition coupled with an easy induction. 

The orthogonal and unitary cases are proved roughly in the same way. In 
section 4 we reduce the support to the singular set. Here the main difference 
is that we use Harish- Chandra descent directly on the group. Note that 
some Levi subgroups have components of type GL so that theorem 2 has to 
be assumed. Finally in section 5 we consider the case of a distribution with 
singular support; the proof follows the same line as in section 3. 

We systematically use two classical results : Bernstein's localization prin- 
ciple and a variant of Frobenius reciprocity which we call Frobenius descent. 
For the convenience of the reader they are both recalled in a short Appendix. 

Similar theorems should be true in the archimedean case. A partial result 
is given by |A-G-S] . 

Let us add some comments on the Theorems themselves. First note that 
Theorem 1' gives an independent proof of a well known theorem of Bernstein: 
choose a basis ei, . . . , of V^, add some vector eo of W to obtain a basis of 
W and let P be the isotropy of eo in GL(l^). Then Theorem B of |Ber] 
says that a distribution on GL(Vr) which is invariant under the action of 
P is invariant under the action of GL(W^). Now, by Theorem 1' such a 
distribution is invariant under the adjoint action of the transpose of P and 



4 



the group of inner automorphisms is generated by the images of P and its 
transpose. We thus get an independent proof of Kirillov conjecture in the 
characteristic 0, non archimedean case. 

The occurrence of involutions in multiphcity at most one problems is 
of course nothing new. The situation is fairly simple when all the orbits 
are stable by the involution thanks to Bernstein's localization principle and 
constructibility theorem ( [B-Zj . [G-Kj ). In our case this is not true : only 
generic orbits are stable. Non stable orbits may carry invariant measures 
but they do not extend to the ambient space (a similar situation is already 
present in |Ber] ). 

An illustrative example is the case n = 1 for GL. It reduces to F* acting 
on as {x,y) i— > {tx,t~^y). On the x axis the measure d*x = dx/\x\ is 
invariant but does not extend invariantly. However the symmetric measure 



does extend. 

As in similar cases (for example [J-Rj ) our proof does not give a simple 
explanation of why all invariant distributions are symmetric. The situation 
would be much better if we had some kind of density theorem. For example 
in the GL case let us say that an element (X, v, v*) of g © © V* is regular 
if (f , Xv, . . . X'^~^v) is a basis of V and {v* , . . . , *X"~^t;*) is basis of V* . The 
set of regular elements is a non empty Zariski open subset; regular elements 
have trivial isotropy subgroups. The regular orbits are the orbits of the 
regular elements; they are closed, separated by the invariant polynomials 
and stable by the involution (see |R-S-1] ). In particular they carry invariant 
measures which, the orbits being closed, do extend and are invariant by the 
involution. It is tempting to conjecture that the subspace of the space of 
invariant distributions generated by these measures is weakly dense. This 
would provide a better understanding of Theorem 2. Unfortunately if true 
at all, such a density theorem is likely to be much harder to prove. 

Assuming multiplicity at most one, a more difficult question is to find 
when it is one. Some partial results are known. 

For the orthogonal group (in fact the special orthogonal group) this ques- 
tion has been studied by B. Gross and D.Prasad ( |G-P] . [Pra-2] ) who for- 
mulated a precise conjecture. An up to date account is given by B. Gross and 
M.Reeder ( |G-R] ) . In a different setup, in their work on "Shintani" functions 
A.Murase and T.Sugano obtained complete results for GL{n) and the split 
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orthogonal case but only for spherical representations ( |K-M-S] . |M-Sj ). Fi- 
nally we should mention, Hakim's publication [H], which, at least for the 
discrete series, could perhaps lead to a different kind of proof. 
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1. Theorem 2(2') implies Theorem l(l') 

A group of type Ictd is a locally compact, totally discontinuous group which is 
countable at infinity. We consider smooth representations of such groups. If 
(vr, i?7r) is such a representation then {7i*,E*) is the smooth contragradient. 
Smooth induction is denoted by Ind and compact induction by ind. For 
any topological space T of type Ictd, S(T) is the space of functions locally 
constant, complex valued, defined on T and with compact support. The 
space S'{T) of distributions on T is the dual space to S{T). 

Proposition 1.1 Let M be a Ictd group and N a closed subgroup, both uni- 
modular. Suppose that there exists an involutive anti- automorphism a of M 
such that <j{N) = N and such that any distribution on M, biinvariant under 
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A^, is fixed by a. Then, for any irreducible admissible representation tt of M 



dim (HomA/(m(ij^(l), tt)) x dim (Rom M{ind^ {1),tt*)) < 1. 



This is well known (see for example |Pra-2] ). 

Remark. — There is a variant for the non unimodular case; we will 
not need it. 

Corollary 1.1 Let M be a Ictd group and N a closed subgroup, both uni- 
modular. Suppose that there exists an involutive anti- automorphism a of M 
such that c3"(iV) = and such that any distribution on M, invariant under 
the adjoint action of N, is fixed by a. Then, for any irreducible admissible 
representation n of M and any irreducible admissible representation p of N 



Proof. Let M' = M x N and A^' be the closed subgroup of M' which is the 
image of the homomorphism n \—>- {n, n) of into M. The map (m, n) i— »• 
mn~^ of M' onto M defines a homeomorphism of M' /N' onto M. The 
inverse map is m i— > (m, 1)A^'. On M' /N' left translations by A^' correspond 
to the adjoint action of A^ onto M. We have a bijection between the space of 
distributions T on M invariant under the adjoint action of A^ and the space 
of distributions S on M' which are biinvariant under A^'. Explicitly 



Suppose that T is invariant under a and consider the involutive anti-automorphism 
a' of M' given by a'{m,n) = {a{m),a{n)). Then 



Using the invariance under a and for the adjoint action of A^ we get 



dim (Homjv(7r|Ar, p*)) x dim (HomAr((7r*)|jv, p)) < 1. 




{SJoa') = {T, [ fia{n)aim),a{n))dn) 



J N 




{SJ). 
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Hence S is invariant under a'. Conversely if S is invariant under a' the same 
computation shows that T is invariant under a. Under the assumption of 
the corollary we can now apply Proposition 1-1 and we obtain the inequality 

dim (}iomM'{ind^',{l), n ® p) j x dim (B.omM'{ind^', {1), ir* <Si p*)j < 1. 

We know that Ind^' (1) is the smooth contragredient representation of indf^',(l); 
hence 

B.omM'{ind^, (1), vr* (g) p*) ^ HomM'(vr ® p, Ind^, (1)). 
Frobenius reciprocity tells us that 

HomA//(7r (g) p, Ind^, (1)) ^ HomAr/((7r ® p)\n', !)• 

Clearly 

HomAr/((7r (g)p)|Ar,, 1) ^ HomAr(p, (vTiTv)*) ^ HomAr(7r|Ar,p*). 

Using again Frobenius reciprocity we get 

HomAr(p, (ttiat)*) ^ B.omM{ind^{p),7i*). 
In the above computations we may replace p by p* and vr by vr*. Finally 
RomM'{ind^',{l),TT* ^ p*) ^ Homjv(p, (tti^v)*) 

^ HomAr(7r|Ar,p*) 

^ HomA/(m(ijlf(p),7r*). 
B.omM'{ind^',{l), vr ® p) ^ HomAr(p*, {{tt*)\n)*) 

^ HomAr((7r*)|Ar,p) 
^ HomA/(m(ijlf(p*),7r). 

□ 

Going back to our situation and keeping the notations of the introduction 
consider first the case of the general linear group. We take M = GL{W) and 
N = GL{V). Let E.„ be the space of the representation tt and let E* be 
the smooth dual (relative to the action of GL(W)). Let Ep be the space 
of p and E* be the smooth dual for the action of GL{V). We know ( |B-Z] 
section 7), that the contragredient representation vr* in E* is isomorphic to 
the representation g 7r(*(7~^) in E^^. The same is true for p*. Therefore an 
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element of Hom7v(7r|7v, p*) may be described as a linear map A from i?^ into 
Ep such that, for G 

Anig) = pC9-')A. 

An element of HomAr((7r*)|jv, p) may be described as a linear map A' from 
Et, into Ep such that, for G 

A'nCg-') = Pig) A'. 

We have obtained the same set of linear maps: 

HomAr((7r*)|Ar,p) ^ HomAr (7r| at , p*) . 

We are left with 2 possibilities: either both spaces have dimension or they 
both have dimension 1 which is exactly what we want. 

From now on we forget Theorem 1 and prove Theorem 2. 

Consider the orthogonal/unitary case, with the notations of the intro- 
duction. In Chapter 4 of |M-V-W] the following result is proved. Choose 
6 G GLfiW) such that {6w,Sw') = {w',w). If n is an irreducible admissible 
representation of M, let n* be its smooth contragredient and define tt^ by 

Then and vr* are equivalent. We choose 5 = 1 in the orthogonal case 
D = F. In the unitary case, fix an orthogonal basis of W, say ei, . . . , e^+i, 
such that 62, ... , Cn+i is a basis of V; put (cj, Cj) = Oj. Then 

Define S by 
Note that 6^ = 1. 

Let -Ett be the space of vr. Then, up to equivalence, tt* is the representation 
m \—>- 7r(5m5^^). If p is an admissible irreducible representation of G in a 

vector space Ep then an element A of Hom ^71"*^,, p^ is a linear map from i?^ 

into Ep such that 

yl7r(5(75-') = 7r(^?)A g E G. 
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In turn the contragredient p* of p is equivalent to the representation g ^ 
p{6g6^^) in Ep. Then an element B of Horn (ttig, p*) is a hnear map from E^,- 
into Ep such that 

B7r{g)=pi6g6-')B, g e G. 
As 5^ = 1 the conditions on A and 5 are the same: 

Hom (7r*g, p) ^ Hom (ttig, p*) ■ 

However, assuming Theorem 2', by Corollary 1-1 we have 

dim^Hom (7r*Q, p) j x dim^Hom (ttig, p*) j < 1. 

so that both dimensions are or 1. Replacing p by p* we get Theorem 1'. 
From now on we forget about Theorem 1'. 



2. Reduction to the singular set : the GL(n) case 

If is a topological group of type Ictd, acting continuously on a topological 
space E of the same type and if x is a continuous character of H we denote 

by S'{E)^'^ the space of distributions T on E such that {T,f{h-^x)) = 
x{h){T, f) for any / G S{E) and any h & H . 

Consider the case of the general linear group. From the decomposition 

= V © Fe we get, with obvious identifications 

End{W) = End(y) ®V®V*®¥. 

Note that End(y) is the Lie algebra q of G. The group G acts on End(PF) by 
g{X, V, V*, t) — {gXg~^, gv, *g~^v*, t). As before choose a basis (ei, . . . , e„) of 

V and let (e*, . . . , e*) be the dual basis of V* . Define an isomorphism u of 

V onto V* by u{ei) = e*. On GL{W) the involution a is h u~^*h^^u. It 
depends upon the choice of the basis but the action on the space of invariant 
distributions does not depend upon this choice. 

It will be convenient to introduce an extension G of G. Let Iso(y, V*) be 
the set of isomorphisms of V onto V*. We define G — GU lso{V, V*). The 
group law, for g,g' e G and u, u' e Iso(V, V*) is 

9^9' = 99' 1 u X g = ug, g X u =^g^^u, u x u' —^u~^u'. 
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Now from W = V (B wc obtain an identification of tlie dual space W* 
with V* © Fe* with (e*, V) = (0) and (e*. e) = 1. Any u as above extends to 
an isomorphism of W onto W* by defining ^(e) = e*. The group G acts on 
GL{W) : 

h ^ ghg-\ h ^\uhu-^), geG,he GL(W), u e Iso(y, V*) 

and also on End (14^) with the same formulas. 

Let X be the character of G which is 1 on G and —1 on Iso(V, V*). Our 
goal is to prove that S'{GL{W))^'^ = (0). 

Proposition 2.1 IfS'{Q ®V®V*f'^ = (0) then S'{GL{W)f'^ = (0). 

Proof. We have End(W^) = ( End(y) ®V®V*)®W and the actioii of G on 
F is trivial thus S'{q® V © V*)^'^ = (0) implies that S'{Eiid{W))^'^ = (0). 
Let T G S'{GL{W))^'^. Let h e GL{W) and choose a compact open neign- 
borhood K of Dct h such that ^ K. For x G End(W^) define ^p{x) = 1 
if Deta; G and 99(0;) = otherwise. Then <^ is a locally constant func- 
tion. The distribution {(p\GL{w))T has a support which is closed in End(l^) 
hence may be viewed as a distribution on End(VF). This distribution be- 
longs to iS'(End(Vr))'^'^ so it must be equal to 0. It follows that T is in 
the neighborhood of h. As h is arbitrary we conclude that T — 0. 

□ 

Our task is now to prove that S'{q ®V ® V*)^'^ = (0). We shall use 
induction on the dimension n of V. The action of G is, for X G 0, w G 

V, V* eV*, g eG,ue lso(V, V*) 

{X, V, V*) ^ {gXg-\ gv,'g-\*), {X, v, v*) ^ {\uXu-')S-'v* , uv). 

The case n = is trivial. 

We suppose that V is of dimension n > 1. assuming the result up to 
dimension n — 1 and for all F. If T G S'{g ®V (B V*)'^'^ we are going to show 
that its support is contained in the "singular set". This will be done in two 
stages. _ 

OnV ®V* let r be the cone {v*, v) = 0. It is stable under G. 

Lemma 2.1 The support ofT is contained in q xT. 
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Proof. For {X,v,v*) e g®V ®V* put q{X,v,v*) = {v*,v). Let Q be the 
open subset q 0. We have to show that S'{Q)'-^'^ = (0). By Bernstein's 
locahzation principle (Corollary 6-1) it is enough to prove that, for any fiber 

= g-i(t), t ^ 0, one has = (0). 

G acts transitively on the quadric {v*,v) = t. Fix a decomposition V = 
¥e®Vi and identify V* = ¥e*®V{ with {e*, e) = 1. Then (X, e, te*) e ilt and 
the isotropy subgroup of {e,te*) in G is, with an obvious notation G„_i. By 
Frobenius descent (Theorem 6-2) there is a linear bijection between S'{Qt)'~^'^ 
and the space 5'(s)^"-i'^ and this last space is (0) by induction. 

□ 

Let 3 be the center of q that is to say the space of scalar matrices. Let 
Af C [q, q] be the nilpotent cone in q. 

Lemma 2.2 The support of T is contained in ^ x x T. 

Proof. We use Harish- Chandra's descent. For X G g let X = -|- X„ 
be the Jordan decomposition of X with X^ semisimple and X„ nilpotent. 
This decomposition commutes with the action of G. The centralizer Zg{X) 
of an element X G g is unimodular ( |Sp-St| page 235) and there exists an 
isomorphism uofV onto V* such that *X = uXu~^ (any matrix is conjugate 
to its transpose). It follows that the centralizer Zq{X) of X in G, a semi 
direct product of Zg{X) and 5*2 is also unimodular. 

Let E be the vector space of monic polynomials, of degree n , with co- 
efficients in F. For p & E, let Qp be the set of all X G g with characteristic 
polynomial p. Note that Qp is fixed by G. By Bernstein localization principle 
(Corollary 6-1) it is enough to prove that if p is not (T — A)" for some A then 
S'iQp xVx V*f'^ = (0). 

Fix p. We claim that the map X X^ restricted to Qp is continuous. 
Indeed let F be a finite Galois extension of F containing all the roots of p. 
Let 

pio = ri(e - kr 

1 

be the decomposition of p. Recall that if X G 0p and Vi = Ker(X — A/)"'* 
then V = (BVi and the restriction of X^ to Vi isjhe multiplication by Aj. 
Then choose a polynomial R, with coefficients in F such that for all i, R is 
congruent to Aj modulo (^ — Aj)"' and -R(O) = to respect the tradition. 
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Clearly Xg = R{X). As the Galois group of F over F permutes the Aj we 
may even choose R G F[^]. This implies the required continuity. 

There is only one semi-simple orbit jp in Qp and it is closed. We use 
Frobenius descent for the map {X, v, v*) t— > Xg from Qp x V x V* to 7p. 

Fix a G 7p ; its fiber is the product of V Q) V* by the set of nilpotent 
elements which commute with a. It is a closed subset of the centralizer 
= '^gia) of a in q. Let M = Zda) and M = Z^{a). 

Following ( |Sp-St| ) let us describe these centralizers. Let P be the min- 
imal polynomial of a ; all its roots are simple. Let P = Pi ... be the 
decomposition of P into irreducible factors, over F. Then the Pi are two 
by two relatively prime. If Vi = KerPj(a), then V = ®Vi and V* = Q)V*. 
An element x of G which commutes with a is given by a family {xi, . . . ,Xr} 
where each linear map from VitoVi, commuting with the restriction of 

a to Vi. Now F[^] acts on V^, by specializing ^ to a\Vi and Pj acts trivially so 
that, if Fj = F[^]/ (Pj), then Vi becomes a vector space over Fj. The F— linear 
map Xi commutes with a if and only if it is Fj— linear. 

Fix i. Let ^ be a non zero F— linear form on Fj. If Vj G Vj and f j G V* then 
A (Af j, f j) is an F— linear form on Fj, hence there exists a unique element 
S{vi,v'j) of Fj such that (Afj,fj) = i {XS{vi,v'j)). One checks trivially that 
S is Fj— linear with respect to each variable and defines a non degenerate 
duality, over Fj between Vi and V*. Here Fj acts on V* by transposition, 
relative to the F— duality (., .), of the action on Vi. Finally if Xi G Endp^l^, 
its transpose, relative to the duality S{., .) is the same as its transpose relative 
to the duality (.,.). 

Thus M is a product of linear groups and the situation (M, V, V*) is a 
composite case, each component being a linear case (over various extensions 
of F). 

Let u be an isomorphism of V j^nto V^uch that *a = uau^-^ and that, 
for each i, u{V) = V*. Then m G M and M = M U uM. 

Suppose that a does not belong to the center of g. Then each Vi has 
dimension strictly smaller than n and we can use the inductive assumption. 
Therefore S'{m ®V ® V"*)^^'^ = (0). However the nilpotent cone in m is 
a closed subset so iS'(A/'m x V" x = (0) which is what we need. 

□ 

If a belongs to the center then M = G and the fiber is (a + Af) xVxV*. 
Therefore we have proved the following Proposition: 
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Proposition 2.2 IfT e S'{g®V®V*)'^'^ then the support ofT is contained 

in ^ X Af xT^ 

IfS'{U X Tf^^ = (0) then S'{q ®V ® V*)^'^ = (0). 

Remark. — Strictly speaking the singular set is defined asjhe set of 
all {X,v,v*) such that for any polynomial P invariant under G one has 
P{X,v,v*) = -P(O). So we should take care of the invariants P{X,v,v*) = 
{v*,XPv) for all p and not only for p = 0. It can be proved, a priori, that 
the support of the distribution T has to satisfy these extra conditions. As 
this is not needed in the sequel we omit the proof. 



In this section we consider a distribution T e S'{Af x F)*^'^ and prove that 
T = 0. The following observation will play a crucial role. 

Choose a non trivial additive character ijj of F. On V (B V* we have the 
bilinear form 



with dvidv* is normalized so that there is no constant factor appearing in 
the inversion formula. 

This Fourier transform commutes with the action of G; hence the (partial) 
Fourier transform T of our distribution T has the same invariance properties 
and the same support conditions as T itself. 

Let Afi be the union of nilpotent orbits of dimension at most i. We will 
prove, by descending induction on i, that the support of any (G, cquivariant 
distribution must be contained in Mi x F. Suppose we already know that, 
for some z, the support must be contained in J\fi x F. We must show that, 
for any nilpotent orbit O of dimension i, the restriction of the distribution 
to O X F is 0. 

li V eV and v* e V* we call X^^^* the rank one map x i— > {v*, x)v. Let 



3. End of the proof for GL(n) 



{ivi,vl), {V2,V2)) ^ {vl,V2) + {V2,vi). 



Define the Fourier transform by 




ux{X,v,v*) = {X + XX^^^.,v,v*), {X,v,v*) eQxT, A G F. 
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Then ux is a one parameter group of homeomorphisms of g x F and note 
that [g, 0] X r is invariant. The key observation is that ux commutes with 
the action of G . Therefore the image of T by ux transforms according to 
the character % of G. Its support is contained in [q, q] x T and hence must 
be contained in jV x F and in fact in A/i x F. This means that if {X, v, v*) 
belongs to the support of T then, for all A, {X + \Xy^^*,v,v*) must belong 
to MiXT. 

The orbit O is open in Mi. Thus ii X E O the condition X + XXy^y* G Mi 
implies that, at least for |A| small enough, X + XX^^^* e O. It follows that 
Xv,v* belongs to the tangent space to O at the point X and this tangent 
space is the image of adX. 

Let us call Q{X) the set of all pairs such X^^^* e Im adX. 

Therefore it is enough to prove the following Lemma: 

Lemma 3.1 Let T e S'{0 xV x V*)^^'>^. Suppose that the support ofT and 
ofT are contained in the set of triplets {X,v,v*) such that {v,v*) e Q{X). 
Then T = 0. 

Note that the trace of X^^^* is {v*, v) and that e Im adX implies that 
its trace is 0. Therefore Q{X) is contained in F. 

We proceed in three steps. First we transfer the problem ioV®V* and 
a fixed nilpotent endomorphism X. Then we show that if Lemma 3-1 is true 
for (VI, Xi) and (V2, X2) then it is true for the direct sum (Vl © ^2, Xi ©X2). 
Finally using the decomposition of X in Jordan blocks we are left with the 
case of a principal nilpotent element for which we give a direct proof, using 
Weil representation. 

Consider the map (X, v-^ X from O x V x V* onto O. Choose 

X ^ O and let G (resp G) be the stabilizer in G (resp. in G) of an element 
X oi O ] both groups are unimodular, hence we may use Frobenius descent 
(Theorem 6-2). 

Now we have to deal with a distribution, which we still call T, which 
belongs to S'iV © V*)^'^ such that both T and its Fourier transform are 
supported by Q{X). Let us say that X is nice if the only such distribution 
is 0. We want to prove that all nilpotent endomorphisms are nice. 

Lemma 3.2 Suppose that we have a decomposition V = Vi © V2 such that 
XiVi) C Vi- Let Xi he the restriction of X to V^. Then if Xi and X2 are 
nice, so is X. 
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Proof of Lemma 3.2. Let Q{X) be the set of pairs {v^v*) such that X^^y* 
belongs to the image of adX. Let G Q{X) and choose A G such 

that Xy^y* = [^4, X]. Decompose v = vi+ V2, v* = vl + V2 and put 



Writing as a 2 by 2 matrix and looking at the diagonal blocks one gets 
that Xv^^y* — [Ai^i,Xi]. This means that 



For i = 1,2 let Ci be the centralizer of Xj in GLiVi) and Cj the corresponding 
extension by 82. Let T be a distribution as above and let Lp2 G 5(V2 © Vg*). 
Let Ti be the distribution on Vi Q) V{ defined by ipi 1— > {T,ipi ® 992) • The 
support of Ti is contained in Q{Xi) and Ti is invariant under the action of 
Ci- We have 



Here <^i(f 1, v^) = (pi{—vi, — f *). By assumption the support of T is contained 
in Q{X) so that the support of Ti is supported in —Q{Xi) = Q{Xi). Because 
(Xi) is nice this implies that Ti in invariant under Ci. Imbedding Ci into 
C we get that T is invariant under Ci. Similarly it is invariant under C2. 
However the subgroup Ci x (72 of (7 is not contained in C so that T must be 
invariant under C and hence must be 0. 



Decomposing X into Jordan blocks we still have to prove Lemma 3-1 for 
a principal nilpotent element. We need some preliminary results. 

Lemma 3.3 The distribution T satisfies the following homogeneity condi- 
tion: 



Proof of Lemma 3.3. We use a particular case of Weil or oscillator represen- 
tation. Let E he a vector space over F of finite dimension m. To simplify 
assume that m is even. Let g be a non degenerate quadratic form on E and 
let b be the bilinear form 




Q(X) c Q(Xi) X Q(X2). 



{Ti,ipi) = (Ti,^i) = (T,^(g)(^2) = (T,9?i (g)^). 



□ 



{T,f{tv,tv*)) = \tnT,f{v,v*)). 



b{e,e')^q{e + e')-q{e)-q{e'). 
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Fix a continuous non trivial additive character ip of F. We define the Fourier 
transform on E by 

/(e')= / f{e)me,e'))de 



J E 

where de is the self dual Haar measure. 

There exists ( |R-S-2j ) a representation vr of S'L(2,F) in S{E) such that: 



1 u 
^' 



^)/(e) 



i^{uq{e))f{e) 



TT 



- 



The 7(tg) are complex numbers of modulus 1. In particular if {E, q) is a sum 
of hyperbolic planes these numbers are all equal to 1. 

We have a contragredient action in the dual space S\E). 

Suppose that T is a distribution on E such that T and T are supported 
by the isotropic cone g(e) = 0. This means that 

(T,vr(j ^)/) = (T,/), (f,7r(j ^)/) = (f,/). 



Using the relation 



(T,¥,) = (T,7(g)7r( Q )/) 



1 



the second relation is equivalent to 



The matrices 



and , u e¥ 



I r \u 1 



generate the group S'L(2,F). Therefore the distribution T is invariant by 
SL{2,¥). In particular 

{TJ{te)) = ^-^\t\-^/'{TJ) 
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and T = j{q)T. 

Remark. — For m even j{tq)/j{q) is a character and there do exist 
non zero distributions invariant under SL{2,¥). In odd dimension we get 
a representation of the 2-fold covering of SL{2,¥) and we obtain the same 
homogeneity condition. However j{tq)/j{q) is not a character; hence the 
distribution T must be 0. 

In our situation we take E — V ®V* and q{v, v*) — {v*, v). Then 

The Fourier transform commutes with the action of G. Both T and T are 
supported by Q{X) which is contained in F. As ^{tq) — 1 this proves the 
Lemma and also that T = T. □ 

Remark. — The same type of argument could have been used for the 
quadratic form Tr{XY) on sl{V) = This would have given a short 

proof for even n and a homogeneity condition for odd n. 

Now we find Q{X). 

Lemma 3.4 If X is principal then Q{X) is the set of pairs {v,v*) such that 
forO<k<n, {v*,X^v) = 0. 

Proof of Lemma 3.4- Choose a basis (ei, . . . , e„) of V such that Xei = and 
— for j > 2. Consider the map A XA — AX from the space of 
n by n matrices into itself. A simple computation shows that the kernel of 
this map, that is to say the Lie algebra c of the centralizer C, is the space of 
polynomials (of degree at most n — 1 ) in X. It is of dimension n. The image 
is of codimension n and calling bij the coefficients of an n by n matrix, a set 
of independent equations for this image is 

n—r 

^bj+r,j = 0, r = 0, ...n-1. 

Let (e^, . . . , e*) be the dual basis. Call Xi, . . . ,Xn the coordinates of v and 
(x^, . . . , x*) the coordinates of v*. The matrix of X^,^* is then given by 
bi,j — XiX* and we get the lemma. 

□ 

End of the proof of Lemma 3.1. For X principal, we proceed by induction 
on n. Keep the above notations. The centralizer C of X is the space of 
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polynomials (of degree at most n — 1 ) in X with non zero constant term. 
In particular the orbit Q of e„ is the open subset a;„ ^ 0. We shall prove 
that the restriction of T to Q x V* is 0. Note that the centralizer of e„ in 
C is trivial. By Frobenius descent (Theorem 6-2), to the restriction of T 
corresponds a distribution R on V* with support in the set of v* such that 
(e„, V*) e Q{X). By the last Lemma this means that i? is a multiple aS of the 
Dirac measure at the origin. The distribution T satisfies the two conditions 

{T,f(v,v*)) = {T,f(tv,r\*)) = \tr{T,f(tv,tv*))- 

therefore 

{TJ{v,t\*) = \tr{TJ{v,v*)). 
Now T is recovered from R by the formula 

{TJ{v,v*)= f {RJ{cen,'c-'v*)dc^a f f{cen,0)dc, feS(nxV*). 
Jc Jc 

Unless a = this is not compatible with this last homogeneity condition. 

Exactly in the same way one proves that T is on V x ^2* where Q* is 
the open orbit 7^ of C in y*. The same argument is vahd for T (which 
is even equal to T . . . ) . 

If n = 1 then T is obviously 0. If n > 2 then there exists a distribution 
T' on 

© Fe* 

l<j<n 

such that, 

5^^=o dxi (g) 5^*=o (8) d<. 

Let u be the isomorphism of V onto V* given by u{ej) = e'^^^^j. Recall that 
it acts on Q X V X V* hj {X,v,v*) 1-^ {\uXu~^),^ u~^v* ,uv). It belongs to 
C but not to C so it must transform T into —T. 

The case n — 1 has just been settled. If n = 2 in the above formula 
T' should be replaced by a constant. The constant must be if we want 
u{T) = -T. If n > 2 let 

V' = (^©r'^ei)/Fei 

and let X' be the nilpotent endomorphism of V defined by X. We may con- 
sider T' as a distribution onV^V'* and one easily checks that, with obvious 
notations, it transforms according to the character x of the the centralizer 
C of X' in G'. By induction T' = 0, hence T = 0. 

□ 
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4. Reduction to the singular set: the orthogonal and 
unitary cases 

We now turn our attention to the unitary case. We keep the notations of the 
introduction. In particular W — V®'Deisa vector space over D of dimension 
n+1 with a non degenerate hermitian form (., .) such that e is orthogonal to 
V. The unitary group G of V is embedded into the unitary group M of W. 
Let A be the set of all bijective maps u from V to V such that 

U{vi+V2) ^U{vi) +U{V2), u{Xv)^Xu{v), {u{vi),u{v2)) ^ {vi,V2). 

An example of such a map is obtained by choosing a basis ei, . . . , e„ of V 
such that (eijCj) e F and defining 

Any u E A is extended to W by the rule u{v + Ae) = u{v) + Ae and we define 
an action on GL(W) by m i— > um~^u~^. The group G acts on GL{W) by 
the adjoint action. 

Let G be the group of bijections of GL{W) onto itself generated by the 
actions of G and A. It is a semi direct product of G and 5*2. We identify 
G to a subgroup of G and A to a. subset. When a confusion is possible we 
denote the product in G with a x . 

We define a character x of G by x{g) — I for g e G and x(ii) = —1 for 
ueG\G. Our overall goal is to prove that S'(M)^'^ = (0). 

Let G act on G x y as follows: 

g{x, v) = {gxg~^, g{v)), u{x, v) = {ux~^u~^, —u{v)), g E G,u E A, x E G,v E V 
Our first step is to replace M by G x l^. 

Proposition 4.1 Suppose that for any V and any hermitian form S'{G x 
T/)G,x ^ (0)^ then S\Mf'^ = (0). 

Proof We have in particular S'{M x W)^'^ = (0). Let Y be the set of 
all {m,w) such that {w,w) — (e,e); it is a closed subset, invariant under 
M, hence S'{Y)^'^ = (0). By Witt's theorem M acts transitively on L = 
{w\{w,w) = (e,e)}. Wc can apply Frobenius descent (Theorem 6-2) to the 
map (m, w) ^-^ w of Y onto F. The centralizer of e in M is isomorphic to G 
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acting as before on the fiber M x {e}. We have a hnear bijection between 
S'{M)^-^ and therefore S'{M)^-^ = (0). 

□ 

The proof that S'{G x V)'^'^ = (0) is by induction on n. If q is the Lie 
algebra of G we shall prove simultaneously that S'{q x V)'^'^ = (0). In this 
case G acts on its Lie algebra by the adjoint action and for m G G \ G one 
puts, for X G 0, u{X) = —uXu^^. 

The case n = is trivial so we may assume that n > 1. If T G iS'(Gx V^)*^'^ 
in this section we will prove that the support of T must be contained in the 
"singular set". 

Let Z (resp. 3) be the center of G (resp. q) and U (resp. N) the (closed) 
set of all unipotent (resp. nilpotent) elements of G (resp. g). 

Lemma 4.1 IfT G S'{G x \/)^'^ (resp. T G S'{q x V)^^>^) then the support 
ofT is contained in ZlA x V (resp. 3 x A/" x y). 

This is Harish-Chandra's descent. We first review some facts about the 
centralizers of semi-simple elements, following |Sp-St . 

Let a G G, semi-simple; we want to describe its centralizer M (resp. M) 
in G (resp. in G) and to show that S'{M x F)^'^ = (0). 

View D— linear endomorphism of V and call P its minimal poly- 

nomial. Then, as a is semi-simple, P decomposes into irreducible factors 
P = Pi . . . Pr two by two relatively prime. Let Vi = KerPj(a) so that 
V = ©Vi. Any element x which commutes with a will satisfy xVi C 
for each i. For 

RiO = rfo + ■ ■ ■ + d^r, dod^ ^ 

let 

^*(0 = ^r + ---+^- 

Then, from aa* = 1 we obtain, if m is the degree of P 

{Pia)v,v') = {v,a-'^P*{a)v') 

(note that the constant term of P can not be because a is invertible). It 
follows that P*(a) = so that P* is proportional to P. Now P* = P* . . . P*; 
hence there exists a bijection r from {1, 2, . . . , r} onto itself such that P* is 
proportional to Pr(j)- Let rrii be the degree of Pj. Then, for some non zero 
constant c 

= {Pi{a)vi, vj) = {vi, a-'"^^P*{a)vj) = c{vi, a~"'^P^(^i){a)vj) , Vi G Vi, Vj G Vj. 
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We have two possibilities. 

Case 1: r(i) = i. The space Vi is orthogonal to Vj for j 7^ i; the restric- 
tion of the hermitian form to Vi is non degenerate. Let ©j = D[^]/(Pj) and 
consider Vi as a vector space over through the action {R{^),v) 1— >• R{a)v. 
As a\Vi is invertible, ^ is invertible modulo {Pi); choose rj such that ^r] — 1 
modulo (Pj). Let (7j be the semi-linear involution of Dj, as an algebra over 



Let ¥i be the subfield of fixed points for ai. It is a finite extension of F, 
and Dj is either a quadratic extension of Fj or equal to Fj. There exists a 
D-linear form £ 7^ on Dj such that i{ai{d)) = ai{i{d)) for aU d e Dj. Then 
any D— linear form L on Dj may be written as d ^ i{\d) for some unique 
A e Dj. 

If v.v' G Vj then d ^ {d{a)v,v') is D— linear map on D,; hence there 
exists S{v, v') e Dj such that 



One checks that S* is a non degenerate hermitian form on Vj as a vector space 
over Dj. Also a D— linear map Xi from Vi into itself commutes with Oj if and 
only if it is Dj-linear and it is unitary with respect to our original hermitian 
form if and only if it is unitary with respect to S. So in this case we call 
Gi the unitary group of S. It does not depend upon the choice of £. As no 
confusion may arise, for A G Dj we define A = o-j(A). 

We choose an Fj— linear map -Uj from Vj onto itself, such that Ui{Xv) = 
Xu{v) and S {ui{v) , Ui{v')) = S{v,v'). Then because of our original choice of 
i we also have {ui{v) , Ui{v')) — {v,v'). Note that u{avi)~^u~^ — avi- 

Case 2. Suppose now that j = T{i) 7^ i. Then V^ © V,- is orthogonal 
to Vk for k 7^ i,j and the restriction of the hermitian form to © V^- is 
non degenerate, both Vj and Vj being totally isotropic subspaces. Choose an 
inverse rj of C, modulo Pj. Then for any P G D[^] 



where P is the polynomial deduced from P by changing its coefficients into 
their conjugate. This defines a map, which we call Uj from Dj onto D^. In a 
similar way we have a map aj which is the inverse of CTj. Then, for A G Dj 
we have {Xvi,Vj) = {vi,ai{X)vj). 



D: 




{d{a)v,v') ^i{dS{v,v')). 



{P{a)vi,Vj) = {vi, P{r){a))vj) , 



Vi G Vi, Vj G Vj 
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View Vi as a, vector space over Dj. The action 

{X,Vj) ai{X)vj 

defines a structure of Dj vector space on Vj. However note that for A G D we 
have (Jj(A) = A so that (7i{\)vj may be different from Xvj. To avoid confusion 
we shall write, for A e Dj 

Xvi — X*Vi and ai{X)vj — X* vj. 

As in the first case choose a non zero D— linear form £ on Dj. For Vi & 
and Vj e Vj the map A i— > {X* Vi, Vj) is a D— linear form on D^; hence there 
exists a unique element S{vi, Vj) e Dj such that, for all A 

{X*Vi,Vj) = £{XS{vi,Vj)). 

The form S* is Dj— bilinear and non degenerate so that we can view Vj as 
the dual space over Dj of the Dj vector space Vj. 

Let {xi,Xj) G Endo{Vi) x Endo{Vj). They commute with (aj,aj) if and 
only if they are Dj-linear. The original hermitian form will be preserved, if 
and only if S{xiVi,XjVj) — S{vi,Vj) for all Vi,Vj. This means that Xj is the 
inverse of the transpose of ,Tj. In this situation we define Gj as the linear 
group of the Dj— vector space Vj. 

Let Ui be a Dj— linear bijection of V^j onto Vj. Then Ui{aVi) = a~^Mj(fj) 
and u^^{avj) = a~^u^^{vj). 

Recall that M is the centralizcr of a in G. Then (M, V) decomposes 
as a "product", each "factor" being either of type (Gj, Vi) with Gj a unitary 
group (case 1) or (Gj, ViXVj) with Gj a general linear group (case 2). Gluing 
together the Wj (case 1) and the (wj, u^^) (case 2) we get an element u G G\G 
such that ua~hir^ — a which means that it belongs^o the centralizer of a in 
G. Finally if M is the centralizer of a in G then (M, V) is imbedded into a 
product each "factor" being either of type (Gj, Vj) with Gj a unitary group 
(case 1) or (Gj, Vi x Vj) with Gj a general linear group (case 2). 

If a is not central then for each i the dimension of Vj is strictly smaller than 
n and from the result for the general linear group and the inductive assump- 
tion in the orthogonal or unitary case we conclude that S'{M x V)^'^ — (0). 

Proof of Lemma 4-1- in the group case. Consider the map g ^ Pg where 
Pg is the characteristic polynomial of g. It is a continuous map from G into 
the set of polynomials of degree at most n. Each non empty fiber T is stable 
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under G but also under G\G. Bernstein's localization principle tells us that 
it is enough to prove that S'{J^ x V)'^'^ = (0). 

Now it follows from Sp-St] chapter IV that JF contains only a finite 



number of semi-simple orbits; in particular the set of semi-simple elements 
J^s in ^ is closed. Let us use the multiplicative Jordan decomposition into a 
product of a semi-simple and a unipotent element. Consider the map 6 from 
J-' X V onto J-'s which associates to {g,v) the semi-simple part Qs of g. This 
map is continuous (see the corresponding proof for GL) and commutes with 
the action of G. In JF^ each orbit 7 is both open and closed therefore ^"^(7) 
is open and closed and invariant under G. It is enough to prove that for 
each such orbit 5'(^~^(7))*^'^ = (0). By Frobenius descent (Theorem 6-2), if 
a G 7 and is not central, this follows from the above considerations on the 
centralizer of such an a and the fact that 6~^^{a) is a closed subset of the 
centralizer of a in G, the product of the set of unipotent element commuting 
with a by V. Now Qs is central if and only if g belongs to ZU, hence the 
Lemma. For the Lie algebra the proof is similar, using the additive Jordan 
decomposition. 

□ 

Going back to the group if a is central we see that it suffices to prove that 
S'{U X V)'^'^ = (0) and similarly for the Lie algebra it is enough to prove 
that S'{Af X Vf'^ = (0). 

Now the exponential map (or the Cayley transform) is a homeomorphism 
of A/" onto U commuting with the action of G. Therefore it is enough to 
consider the Lie algebra case. 

We now turn our attention to V. Let 

T = {v e V\{v,v) = 0}. 



Proposition 4.2 IfTE S'{Af x \/)*^'^ then the support ofT is contained in 
AfxT. 

Proof. Let 

Tt = {veV\{v,v) = 0}. 

Each Ff is stable by G, hence, by Bernstein's localization principle (Corollary 
6-1), to prove that the support of T is contained in A/" x Fq it is enough to 
prove that, for t ^ 0, S'{Af x Tt)^^^ = (0). 
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By Witt's theorem the group G acts transitively on Tf. We can apply 
Frobenius descent to the projection from A/" x Tt onto F^. Fix a point Vq G F^. 
The fiber is A/" x {vq}. Let Gi be the centralizer of vq in G. We have to show 
that S\M)^^'^ = (0) and it is enough to prove that S\Qf^'^ = (0). 

The vector vq is not isotropic so we have an orthogonal decomposition 

with Vi orthogonal to vq. The restriction of the hermitian form to Vi is non 
degenerate and Gi is identified with the unitary group of this restriction, and 
Gi is the expected semi-direct product with 82- As a Gi— module the Lie 
algebra Q is isomorphic to a direct sum 

where Qi is the Lie algebra of Gi and W a vector space over F of dimension 
or 1 and on which the action of Gi is trivial. The action on Qi © Vi is the 
usual one so that, by induction, we know that S'{gi ® Vi)'^'^'^ — (0). This 
readily imphes that S'{q)^^'^ — (0). 

□ 

Summarizing: we have to prove that S'{J\f x F)*^'^ = (0). 

5. End of the proof in the orthogonal and unitary cases 

We keep our general notations. We have to show that a distribution on A/'x F 
which is invariant under G is invariant under G. To some extent the proof 
will be similar to the one we gave for the general linear group. 

In particular we will use the fact that if T is such a distribution then 
its partial Fourier transform on V is also invariant under G. The Fourier 
transform on V is defined using the bilinear form 

{V1,V2) ^ {vi,V2) + {v2,Vi) 

which is invariant under G. 
For V eV put 

(Pv{x) — {x,v)v, X eV. 
It is a rank one endomorphism of V and {ipy{x),y) = {x, ip^i^y)). 
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Lemma 5.1 i)In the unitary case, for A G D such that A = —A the map 

fx- {X,v) ^ {X + \ipy,v) 

is a homeomorphism of [q, g] x F onto itself which commutes with G. 
a) In the orthogonal case, for A G F the map 

fix : (X, v) {X + XX (f^ + X(pvX, v) 

is a homeomorphism of [q,q] x F onto itself which commutes with G. 

The proof is a trivial verification. 

We now use the stratification of A/". Let us first check that an adjoint 
orbit is stable not only by G but by G. 

Choose a basis ei, . . . , of such that (cj, cj) G F; this gives a conjuga- 
tion u : V = ^i^i ^ = X] ■'■f ^ is endomorphism of V then 
A is the endomorphism v ^ A.{v). The conjugation u is an element of G \ G 
and, as such, it acts on q x V hj {X,v) ^-^ {—uXu~^, —u{v)) = {—X, —v). 
In |M-V-W] Chapter 4 Proposition 1-2 it is shown that for X G g there 
exists an F— linear automorphism a of F such that (a(x), a{y)) = {x, y) (this 
implies that a{\x) = Ax) and such that aXa^^ = —X. Then g = ua & G 
and gXg^^ = —X so that —X belongs to the adjoint orbit of X. Note that 
a G G \ G and as such acts as a{X, v) = {X, —a{v)); it is an element of the 
centralizer of X in G \ G. 

Remark. We need to check this only for nilpotent orbits and this will be 
done later in an explicit way, using the canonical form of nilpotent matrices. 

Let Afi be the union of all nilpotent orbits of dimension at most i. We 
shall prove, by descending induction on i, that the support of a distribution 
T G S'{J\f X F)"^'^ must be contained in M x F. 

So now assume that i > and that we already know that the support of 
any T G iS'(A/' x F)*^'^ must be contained in A/^ x F. Let O he a. nilpotent 
orbit of dimension i; we have to show that the restriction of T to (9 is 0. 

In the unitary case fix A G D such that A = —A and consider, for every 
t G F the homeomorphism Utx] the image of T belongs to iS'(A/' x F)*^'^ so 
that the image of the support of T must be contained in Afi x F. If (X, f) 
belongs to this support this means that X + tXipy G Afi. 

If z = so that A/i = {0} this implies that f = so that T must be a 
multiple of the Dirac measure at the point (0, 0) and hence is invariant under 
G so must be 0. 
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If i > and X E O then as O is open in A/"j, we get that, at least for \t\ 
small enough, X + 1 A^jy G (9 and therefore X(py belongs to the tangent space 
Imad(X) of O at the point X. Define 

Q{X) ^ {v e V\(py e Imad(X)}, X eJ\f, (unitary case). 

Then we know that the support of the restriction of T to O is contained in 



and the same is true for the partial Fourier transform of T on y. 

In the orthogonal case for i — 0, the distribution T is the product of the 

Dirac measure at the origin of g by a distribution T' on V. The distribution 
T' is invariant under G but the image of G in End(V) is the same as the 
image of G so that T' is invariant under G hence must be 0. 

If i > we proceed as in the unitary case, using fix. We define 

Q{X) ^ {v e V\Xipy + ifyX e Im eid{X)}, X eJ\f, (orthogonal case) 

and we have the same conclusion. 

In both cases, for i > 0, fix X G O. We use Frobenius descent for the 
projection map {Y,v) Y of O x V onto O. Let G (resp. G) be the 
stabilizer of X in G (resp. G). We have a linear bijection of S'{0 x F)"-^'^ 
onto S'{Vf'^. 

Lemma 5.2 Let T G S'{V)^'^. IfT and its Fourier transform are supported 
in Q{X) then T = 0. 

Let us say that a nilpotent element X is nice if the above Lemma is true. 

Suppose that we have a direct sum decomposition V = Vi®V2 such that 
Vi and V2 are orthogonal. By restriction we get non degenerate hermitian 
forms (., .)i on Vi. We call the unitary group of (., Qi its Lie algebra 
and so on. Suppose that XiVi) C so that Xi — X\Vi is a nilpotent element 



Proof of Lemma 5.3. We claim that Q{X) C Q{Xi) x Q{X2). Indeed if 



{{X,v)\XeO,veQ{X)} 



of 0j. 



Lemma 5.3 If Xi and X^ are nice so is X. 
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then from 
we get in particular 

{A-i^iXi, Hi) + (xi^ Ai^iUi) = 



so that Ai^i G Qi. Note that 
[X, A] = 



If Vi e Vi and vj G V,- we define (fiy^^y^ : V^, by (fy^^y^{xi) = {xi,Vi)vj. 

Then, for v = Vi + V2 

Therefore if, for A G g we have if^ = [X,A] then (pv^,Vi = This 
proves the assertion for the unitary case. The orthogonal case is similar. 
The end of the proof is the same as the end of the proof of Lemma 3-2. 

□ 

Now in both orthogonal and unitary cases nilpotent elements have normal 
forms which are orthogonal direct sums of "simple" nilpotent matrices. This 
is precisely described in |Sp-St] IV 2-19 page 259. By the above Lemma it 
is enough to prove that each "simple" matrix is nice. 

Unitary case. There is only one type to consider. There exists a basis 
ei, . . . , Cn of such that Xei = and Xci = ej_i, i > 2. The hermitian 
form is given by 

(ci, ej) = Oifi + j^n + 1, {a, e„+i_,) = 

with a ^ 0. Note that a = (— Suppose that v G Q{X); for some 
A G we have X(py = XA — AX. For any integer p > 

Tr{XifyXP) = Tr{XAXP - AXp+^) = 0. 

Now Tr{(pyXP) = {X^VjV) Let v = J^^i^i- Hence 

n~p n—p 

(X^v^v) = '^Xi+p{ei,v) = ^(-l)""*axi+px„+i_,; = 0. 
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For p — n — 1 this gives = 0. For p = n — 2 we get nothing new but for 
p = n — 3 we obtain ,t„_i = 0. Going on, by an easy induction, we conclude 
that Xi = Oifi>{n + l)/2. 

If n = 2p + 1 is odd put Vi = ©^De^, Vo = Dcp+i and V2 = eJ^V^e^. If 
n = 2p is even put Vi = ©fDcj, Vo = (0) and V2 = ©p^^Dei. In both cases 
we have V — Vi®Vo®V2. We use the notation v — V2 + V0 + V1 

The distribution T is supported by Vi. Call Si the Dirac measure at on 
Vi. Then we may write T = U © 5o ® ^2 with U G S'{Vi). The same thing 
must be true of the Fourier transform of T. Note that f/ is a distribution 
on V2, that 62 is a Haar measure dvi on Vi and that, for n odd 60 is a Haar 
measure dvo on Vq. So we have T — dvi®U if n is even and T = dvi®dvQ®U 
if n is odd. In the odd case this forces T = 0. In the even case, up to a scalar 
multiple the only possiblity is T = dvi ® 82- 

Let 

a : ^Y^XiCi y^(-l)'xjej. 

Then a G G\G. It acts on g by F 1— > — aya"-*^ and in particular —aXa~^ — X 
so that a G C\C. The action on K is given hj v —a{v). It is an involution. 
The subspace Vi is invariant and so dvi is invariant. This implies that T is 
invariant under C so it must be 0. 

Orthogonal case. There are two different types of "simple" nilpotent 
matrices. 

The first type is the same as the unitary case, with a = 1 and thus n 
odd but now our condition is that Xcpy + (p^X = [X, A] for some A E Q. As 
before this implies that Tr{(pyX'^) — but only for g > 1. Put n — 2p + 1; 
we get Xj — for j > p + 1. Decompose V as before: V — Vi®Vo®V2. Our 
distribution T is supported by the subspace ^2 = so we write it T — U <S>S2 
with U G iS'(Vi©Vo). This is also true for the distribution T so we must have 
U = dvi © R with R a distribution on Vq. Finally T = dv\ © © ^2. Now 
—Id G C and T is invariant under C so that R must be an even distribution. 
On the other end the endomorphism a olV defined by a(ej) = (— l)*~^~^ei 
belongs to Cand aXa"^ — —X and u : {X, v) 1— > {—X, —v) belongs to G\ G. 
The product a * u of a and u in G belongs to C \ C Clearly T is invariant 
under a * m so that T is invariant under C so it must be 0. 

The second type is as follows. We have n = 2m, an even integer and a 
decomposition V = E (B F with both E and F of dimension m. We have a 
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basis ei, . . . , of £^ and a basis /i, . . . , of F such that 

^j) = fj) = 

and 

(eijj) = Oifi + j + 1 and {ei, fm+i-i) = {-ir~'- 

Finally X is such that Xcj = ej_i, Xfi = 

Let ^ be the matrix of the restriction of X to or to F. Write an element 
A e g as 2 by 2 matrix A = (ajj). Then 

Suppose that v e Q{X) and let 

V = e + f with e = ^ XiCi, / = X] 

We get 

where, for example </?e,e is the map /' 1— > (/',e)e from F into F. Thus, for 
some A, 

In this formula, using the basis (cj), (/i) replace all the maps by their matri- 
ces. 

Then, as before, we have Tr((y9e,e^^) = for 1 < g < m — 1. If e' = ^ Xifi 
(the the coordinates of e), then Tr(^''(/9e,e) is (^^e,e'). Thus, as in the 

other cases, we have Xj = for j > n/2 if n is even and j > (n + l)/2 if n is 
odd. The same thing is true for the yi. 

If n = 2p is even, let Vi = ®i<p{^ei ® F/j) and V2 = ®i>p{^ei ® Wfi); 
write ^ = ^1 + ^2 the corresponding decomposition of an arbitrary element of 
V . Let 82 be the Dirac measure at the origin in V2 and dvi a Haar measure 
on Vi- Then, as in the unitary case, using the Fourier transform, we see that 
the distribution T must be a multiple of dvi ® 82. 

The endomorphism a of y defined by a(ei) = (— l)*ei and a(/i) = (— 1)*+^/^ 
belongs to G and aXa~^ — —X. The map u : {Y, v) 1— > (— Y, —v) belongs to 
G\G so that the product a x u in G belongs to C \C. It clearly leaves T 
invariant so that T — 0. 
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Finally if n = 2p + 1 is odd we put Vi = ©i<p(Fej © F/j), Vq = Fcp+i © 
F/p_|_i, V2 = ©j>p+2(Fej © F/j). As in the unitary case we find that T = 
dvi ® i? ® (^2 with R a distribution on Vq. As —Id G C we see that R must be 
even. Then again, define a G G by a(ej) = (— l)*ei and a{fi) = (— and 
consider a * u with u(Y, v) = {—Y, —v). As before a * u G C \ C and leaves 
T invariant so we have to take T = 0. 

□ 

6 Appendix 

We shall state two theorems which are systematically used in our proof. 

If X is a Hausdorff totally disconnected locally compact topological space 
(Ictd space in short) we denote by S{X) the vector space of locally constant 
applications with compact support of X into the field of complex numbers 
C. The dual space S'{X) of S{X) is the space of distributions on X. All the 
Ictd spaces we introduce are countable at infinity. 

If an Ictd topological group G acts continuously on a Ictd space X then 
it acts on S{X) by 

i9f){x) = fig-'x) 

and on distributions by 

i9T)if)=Tig-'f) 

The space of invariant distributions is denoted by S'{X)'^. More generally, 
if X is a character of G we denote by S'{X)'~^'^ the space of distributions T 
which transform according to x that is to say T{f{g~^x)) = x{g)T{f). 
The following result is due to Bernstein [Ber] . section 1.4. 

Theorem 6.1 (Localization principle) Let q : Z ^ T be a continuous 
map between two topological spaces of type Ictd. Denote Zt := q^^it). Con- 
sider S'{Z) as S{T)-module. Let M be a closed subspace of S'{Z) which is 
an S{T)-submodule. Then M = ©^^^(M n S'{Zt)). 

Corollary 6.1 Letq : Z ^ T be a continuous map between topological spaces 
of type Ictd. Let an Ictd group H act on Z preserving the fibers of q. Let n 
be a character of H . Suppose that for any t E T, S'{q~^{t))^'f^ = 0. Then 
S'{Z)^'^' = 0. 

The second theorem is a variant of Frobenius reciprocity. 
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Theorem 6.2 (Frobenius descent) Let a unimodular Ictd topological group 
H act transitively on an Ictd topological space Z . Let (f : E ^ Z be an H- 
equivariant map of Ictd topological spaces. Let x & Z. Suppose that its 
stabilizer Stabnix) is unimodular. Let W be the fiber of x. Let x be a char- 
acter of H. Then 

(i) There exists a canonical isomorphism Fr : S'{E)^'^ [y[/"^stabH{x),x ^ 
(a) For any distribution ^ G S'{E)^'^, Supp(Fr(^)) = Supp(^) fl W. 
(Hi) Frobenius descent commutes with Fourier transform. 

Namely, let W be a finite dimensional linear space over ¥ with a nonde- 
generate bilinear form B. Let H act on W linearly preserving B. 

Then for any ^ E S\Z x W)"^^, we have J^B(Fr(0) = ^t{J^b{.C)) where 
Fr is taken with respect to the projection Z x W Z . 
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